[CAPPL-921] add container retry to clnode, postgres, jd and evm blockchains #1863
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
skudasov
requested changes
Jun 6, 2025
Contributor
skudasov
left a comment
skudasov
left a comment
There was a problem hiding this comment.
We use static ports so we can then apply chaos and break anything without exposing testcontainers-go layer to the test.
Linux systems typically use 32768-60999 for temporary ports, this issue was caused because JD was exposing 42424, maybe we can change the mapping to 125XX for JobDistributor and have this service both testable and without conflicts?
We run strictly one environment per GHA runner, so mapping to 125XX should fix it.
Contributor
Author
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Succesful run: https://github.com/smartcontractkit/chainlink/actions/runs/15484446872/job/43596302667?pr=18037
Below is a summarization created by an LLM (gpt-4-0125-preview). Be mindful of hallucinations and verify accuracy.
Why
The changes improve the robustness and error handling of container start-up processes across various components (blockchain nodes, CL nodes, JD, and PostgreSQL) by introducing retries. This is particularly useful in environments where transient issues may temporarily prevent a container from starting successfully.
What
framework.StartContainerWithRetryfor EVM containers. This change improves error handling by retrying container start-up, reducing the likelihood of failures due to transient issues.framework.StartContainerWithRetry. This modification ensures better stability by attempting to start the container multiple times in case of initial failure.framework.StartContainerWithRetry. Enhances reliability by trying to start the container multiple times if the first attempt fails.framework.StartContainerWithRetry. This adjustment aims to increase success rates of container starts by implementing a retry mechanism.StartContainerWithRetryfunction along withNaiveRetrierand helper functions. These additions provide the capability to retry starting containers with a simple retry strategy, thereby enhancing the resilience of the container start-up process.Below is a summarization created by an LLM (gpt-4-0125-preview). Be mindful of hallucinations and verify accuracy.
Why
The primary goal of these changes is to enhance the stability and reliability of container initialization in integration tests. By incorporating a retry mechanism when starting containers, we address transient issues that may occur due to various reasons such as network delays or resource constraints, ensuring a more robust setup for testing environments.
What
testcontainers.GenericContainercall withframework.StartContainerWithRetryto attempt container start with retries.tc.GenericContainercall withframework.StartContainerWithRetryfor retrying container starts.tc.GenericContainercall withframework.StartContainerWithRetryadding retry logic when starting containers.testcontainers.GenericContainerwithframework.StartContainerWithRetryto implement retry functionality on container start.NaiveRetrierfunction which attempts to restart the container with a simple retry logic.StartContainerWithRetryfunction that tries to start a container with up to 3 retries using provided retriers or a default one.removeContainerfunction to support retry logic by removing existing containers before retrying.